Skip to content

Conversation

rkistner
Copy link
Contributor

@rkistner rkistner commented Sep 16, 2024

Configurable default schema

Let the replicator module specify the default schema, instead of using a DEFAULT_SCHEMA const.

For postgres the 'public' schema is easy. However, for other source databases such as mysql or mongodb, we want to use the database from the connection string as the default, while possibly allowing other databases to be explicitly specified.

Flexible replication id

Let the replicator module compute the replica id for each operation, instead of auto-computing. This can now be persisted as any BSON-compatible value, instead of always converted to a v5 UUID.

Currently, for Postgres, we compute the replica identity as a hash (v5 uuid) from all replica identity columns, after converting to the SQLite format. However, we want the replica identity behavior to match the source database as close as possible, and we lose precision in this way. For example, the old approach would consider _id: ObjectId('66e834cc91d805df11fa0ecb') and _id: '66e834cc91d805df11fa0ecb' as equivalent, while MongoDB considers those as distinct ids. So the new approach allows any BSON-compatible value to be used, and the replicator module can decide what to use.

This does not change anything for Postgres yet.

Implementation notes

  1. SyncRulesBucketStorage instances (BucketStorageFactory.getInstance()) doesn't include a parsed copy of sync rules by default. It is parsed on startBatch() instead, or by calling getParsedSyncRules(). This affects some other APIs as well.
  2. For new-style replica ids, row subkeys now combine the table and replica id in a single UUID. For old-style (postgres) subkeys, this is kept as <table>/<replica id>. We may migrate the Postgres ones to the new smaller format as well in the future.

@changeset-bot
Copy link

changeset-bot bot commented Sep 16, 2024

⚠️ No Changeset found

Latest commit: 2503cba

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@rkistner rkistner marked this pull request as ready for review September 16, 2024 13:39
@rkistner rkistner requested review from Rentacookie and stevensJourney and removed request for Rentacookie September 16, 2024 13:39
@rkistner rkistner merged commit 4fad0b6 into feat/modular-replication-architecture Sep 17, 2024
7 of 9 checks passed
@rkistner rkistner deleted the configurable-default-schema branch September 17, 2024 14:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants